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ABSTRACT 

Two  procedures  for  evaluating  minehunting  sonar  performance  are  presented.  One  for 
detection  with  respect  to  range,  the  other  for  detection  with  respect  to  CPA  or 
athwartships  distance.  Both  procedures  consider  the  trials  results  as  subsets  of 
normally  distributed  data  and  use  fitted  normal  curves  to  calculate  cumulative 
probability  of  detection  curves  which  can  be  then  assessed  against  specification. 
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New  Methods  to  Evaluate  the  Detection 
Performance  of  a  Minehunter 


Executive  Summary 

The  effectiveness  of  a  minehunting  sonar  is  dependent  on  detection  performance  in 
both  along  track  (range)  and  in  across  track  (athwartships)  directions.  Assessment 
against  specification  entails  conducting  many  detection  runs  against  known  targets 
and  statistical  analysis  of  results. 

This  document  proposes  two  new  methods  for  assessing  trials  results  against 
specification.  The  first  method,  used  for  detection  range  trials,  is  regarded  as  an 
improved  methodology  over  a  current  procedure.  The  conclusions  drawn  using  the 
new  methodology  are  more  reliable,  and  better  reflect  the  true  performance  of  the 

sonar. 

The  second  method  is  applicable  for  evaluation  of  detection  width  trials  and  puts  in 
place  a  formal  methodology  for  data  analysis.  The  equal  area  principle,  defined  by  the 
conventional  parameters  A  (swath  width)  and  B  (probability  of  detection),  is  applied  to 
the  processed  data  to  compare  results  to  the  performance  specification. 
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1.  Introduction 

The  detection  performance  of  minehunting  sonars  is  usually  assessed  by  conducting 
detection  trials  against  known  targets  in  a  known  or  measured  environment.  Due  to 
limitations  of  time  and  cost  the  number  of  detection  runs  has  to  be  minimised,  subject 
to  validation  of  statistics,  for  each  trial  objective.  Normally,  25  to  30  runs  are  planned 
for  each  trial  and  conclusions  about  the  sonar  performance  must  be  drawn  from  these 
25  or  30  samples.  How  to  make  reliable  statistical  inference  is  crucial  in  the  trials 

analysis. 

In  this  document,  two  methodologies  are  established  for  evaluating  the  detection 
performance  of  a  minehunting  sonar.  The  first  method  is  appropriate  for  Detection 
Range  Trials  (DRT)  that  are  designed  to  assess  the  detection  performance  as  a  function 
of  range  ahead  of  the  vessel.  The  second  method  is  developed  for  Detection  Width 
Trials  (DWT)  where  the  detection  capability  against  across  track  distance  is  evaluated. 
Implicit  in  the  assessment  trials  is  the  assumption  that  the  results  from  a  limited 
number  of  runs  are  representative  of  the  sonar  performance  m  general.  That  is,  a 
comparatively  limited  number  of  runs  are  representative  and  statistically  valid  samples 
from  the  set  of  all  possible  detection  runs.  Typically  25  to  30  runs  are  conducted  for 
detection  range  trials.  Many  more  runs  are  needed  for  detection  width  trials  to  ensure 
sufficient  samples  at  all  cross  track  range  intervals. 

An  evaluation  methodology  for  detection  range  trials  has  been  described  by  Thompson 
and  Bell  1997  [1].  This  procedure  was  used  effectively  during  trials  of  the  RAN  Mine 
Hunter  Inshore  (MHI).  In  this  paper,  the  methodology  is  improved  by  clarifying 
mathematical  relationships  between  the  detection  range  and  the  probability  of 
detection.  It  is  important  to  note  that  the  probability  of  detection  of  a  target,  and  the 
range  at  which  a  certain  probability  of  detection  is  achieved  are  two  different  concepts. 
In  order  to  compare  trials  results  to  the  performance  specification  it  is  necessary  to 
combine  both  these  factors. 

Thompson  and  Bell  use  a  cumulative  probability  of  detection  against  range  to  quantify 
the  sonar  performance  [1],  however  the  definition  of  probability  of  detection  is  valid 
only  if  there  is  no  failure  in  the  detection  runs  and  the  target  is  detected  on  all  possible 
encounters.  The  procedure  does  not  satisfactorily  account  for  non-detections  (misses) 
as  it  sets  the  range  to  zero  for  runs  where  no  detection  is  made.  The  methodology  is,  by 
this  definition,  insufficiently  robust  to  handle  the  situation  when  one  or  more  detection 
runs  fail  because  the  cumulative  probability  of  detection  was  not  based  on  any  density 
functions. 

The  methodology  presented  here  for  the  detection  range  trials  is  considered  a  more 
accurate  and  reliable  procedure  than  that  of  Thompson  and  Bell  and  as  a  result,  the 
conclusions  drawn  from  the  trials  results  fairer.  This  is  particularly  so  for  a  limited 
number  of  detection  runs  (small  number  of  samples),  and  for  un-indicated  trials  where 
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detection  may  not  be  achieved  in  some  runs.  The  crux  of  the  method  is  that  the 
cumulative  probability  of  detection  is  defined  implicitly  by  the  Cumulative 
Distribution  Function  (cdf)  of  the  ranges  at  which  detections  occur.  It  is  possible 
therefore  to  determine  the  Probability  Density  Function  (pdf)  of  the  detection  range 
and  from  this  the  corresponding  cumulative  distribution  function. 

For  detection  width  trials  a  second  methodology  is  established  for  measurement  of  the 
probability  of  detection  with  respect  to  Closest  Point  of  Approach  (CPA).  The  concept 
is  straightforward,  first  determining  detection  probability  as  a  histogram  with  respect 
to  the  lateral  displacement  of  a  target  from  the  ship's  track,  and  second,  using  curve 
fitting  to  obtain  a  smoothed  curve  that  represents  properties  of  the  sample  population. 
The  area  equal  principle  is  then  used  to  compare  the  result  generated  against  the 
performance  criteria. 

Confidence  intervals  for  both  methodologies  are  addressed  but  as  yet  the  significance 
for  detection  width  trials  is  not  properly  understood. 


2.  Interpretation  of  Trials  Specification 

The  trials  procedure  is  designed  to  demonstrate  that  the  sonar  meets  performance 
criteria  in  accordance  with  the  specification.  Typically,  for  detection  range  trials,  the 
specification  will  require  that  the  sonar  shall  detect,  at  a  certain  probability  of 
detection,  a  given  type  of  target  at  ranges  greater  than  some  defined  distance.  We  need 
to  determine  what  does  this  really  mean  and  how  should  we  interpret  the 
requirement?  For  example,  the  specification  may  indicate  the  sonar  is  capable  of 
detecting  mine  of  target  strength  X  at  ranges  greater  than  500  metres  ( r  >  500  m)  at  a 
probability  of  detection  Pd  =  0.9,  from  10  to  100  metres  depth  over  a  sand  seabed  type. 
This  Pd  is  defined  by  m/N,  where  ra  is  the  number  of  detections  and  N  is  the  number  of 
runs. 

This  specification  can  be  satisfied  by  two  limiting  results  that  define  the  extremes  of 
satisfactory  outcomes.  If  all  runs  score  hits  and  no  misses  occur  (Pd  =  1),  then  at  least 
90%  of  the  detections  must  be  at  ranges  r  >  500  m.  Conversely  if  there  are  10%  misses 
( Pd  =0.9),  then  100%  of  detections  achieved  must  be  at  ranges  r  >  500m.  Generally  the 
result  lies  in  between  these  limits  and  we  have  to  determine  which  parameter  varies, 
r ,  Pd  or  both. 

For  detection  width  trials  the  conventional  parameters  A  and  B,  as  defined  in  the 
NATO  publication  ATP-6  (B)  [2],  are  given  in  the  specification.  Parameter  B  is  the 
height  of  a  trapezoid  or  rectangle  (ie,.  the  probability  of  detection)  and  parameter  A  is 
the  width  of  the  trapezoid  at  one-half  B  or  the  width  of  rectangle  (ie,  detection  width). 
For  example,  the  sonar  specification  may  require  detection  of  specified  targets  at  a 
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probability  of  0.9  over  a  swath  width  of  200  metres.  That  is  B  0.9  for  A  200.  Since  the 
parameters  A  and  B  determine  a  unique  area,  the  methodology  for  determining 
whether  trials  results  meet  the  specification  can  be  based  on  a  comparison  of  areas  as 
described  in  the  following  section. 


3.  Methods  for  Evaluating  Performance 


A  sonar  detection  trial  consists  of  a  number  of  detection  runs  and  each  run  includes  at 
least  one  planned  encounter  between  the  minehunter  sonar  and  a  mine.  If  detection 
fails  (ie,  an  encounter  is  not  marked  by  the  sonar  operator),  the  result  is  a  miss . 

Otherwise,  it  is  a  'hit'. 

The  objective  of  detection  range  and  detection  width  trials  is  to  draw  conclusions  on 
the  actual  distribution  of  hits  and  misses,  and  the  locations  of  those  hits,  from  the 
empirical  (sample)  distributions  obtained  during  the  limited  number  of  detection  runs. 
By  combining  results  from  previous  experiments  and  the  data  from  the  trial,  one  can 
usually  postulate  the  general  nature  of  the  frequency  distribution.  From  probability 
theory  for  independent  trials,  which  is  the  case  here,  the  premise  that  the  possible 
outcomes  in  the  trial  will  occur  equally  often  implies  that  a  large  number  of  runs  will 
yield  the  probability  distribution  for  the  general  case.  Thereafter,  one  can  draw 
conclusions  on  whether  the  specification  is  satisfied. 

One  means  of  assessing  detection  performance  is  to  quantify  the  trials  data  on  a  plot  of 
probability  of  detection  against  range  of  detection.  By  comparing  the  curve  so 
generated  with  the  sonar  specification,  conclusions  on  the  success  or  failure  of  the  trials 
results  can  be  drawn.  This  raises  the  question,  how  to  establish  a  relationship  between 
P  and  detection  range  which  reflects  the  specification  requirement?  Essentially,  a 

cumulative  distribution  function  of  the  detection  range,  scaled  by  a  measured  Pd ,  is 
required. 


3.1  Detection  Probability  With  Respect  to  Range 

The  probability  of  detection  can  be  estimated  by  the  frequency  of  the  hits  against  the 
number  of  encounters.  That  is,  Pd  is  determined  from  the  set  of  possible  encounters. 
We  are  also  interested  in  the  ranges  at  which  hits  occur  since  these  constitute  the  other 
parameter  in  the  detection  vs  range  curve.  This  range  is  also  a  random  variable  defined 
in  this  case  on  the  domain  formed  by  the  subset  of  encounters  corresponding  to  the 
hits.  Hence,  Pd  cannot  be  defined  as  a  simple  function  of  the  range.  Indeed,  we  have  a 
graphic  expression  (figure  1)  showing  the  probability  of  detection  as  a  function  of 
range  for  a  single  ping.  However,  this  probability  is  not  the  Pd  discussed  here.  It  is  a 
conditional  probability  function,  which  indicates  a  probability  that  a  target  at  a  certain 
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range  can  be  detected  by  a  single  ping.  It  cannot  be  integrated  along  the  range  to  obtain 
a  cumulative  Pd  because  the  curve  is  not  a  probability  density  function  (pdf). 


Figure  1.  The  probability  of  detection  as  a  function  of  range  for  a  single  ping. 


To  demonstrate  the  analysis  we  use  the  same  example  given  by  Thompson  and  Bell  for 
a  detection  range  trial  with  a  total  of  20  detection  runs.  A  set  of  20  ranges  is  logged  due 

to  no  misses  (Table  1).  Hence,  both  m  and  N  are  20  and  Pd  =  m/N  =  1.  The  minimum 
and  maximum  ranges  are  157  m  and  507  m. 


Table  1 .  Typical  DRT  log 


Run  Number 

Detection  Range  (m) 

Run  Number 

Detection  Range  (m) 

1 

319 

11 

431 

2 

253 

12 

507 

3 

270 

13 

389 

4 

312 

14 

421 

5 

359 

15 

378 

6 

157 

16 

409 

7 

217 

17 

341 

8 

230 

18 

328 

9 

351 

19 

318 

10 

450 

20 

298 
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A  range  Hass  interval  for  the  analysis  is  chosen  depending  on  how  precisely  the  range 
distribution  is  to  be  analysed.  After  the  class  boundaries  have  been  determined  it  is  a 
simple  matter  to  locate  each  of  the  measured  ranges  in  die  proper  class  interval.  A 11 
ranges  between  the  limits  and  including  the  (closed)  upper  limit  are  placed  in  the  same 
class,  and  die  number  detection  ranges  for  each  class  interval  recorded.  The  raw  data 
are  then  classified.  Table  2  illustrates  the  classification  and  distribution  of  ranges  from 
the  above  detection  range  trial  when  a  50  metre  class  interval  is  selected.  The  interval 
width  can  be  varied  depending  upon  the  number  of  samples.  In  general,  50  -  100 
metres  is  appropriate  to  30  runs. 


Table  2.  Frequency  Table 


Class  Boundaries  (m) 


Numbers 


Frequencies 


50  - 100 
100  - 150 
150  -  200 
200  -  250 
250  -  300 
300  -350 
350  -  400 
400  -  450 
450  -  500 
500  -  550 
550  -  600 
Total 


0 

0 

1 

1 

3 

4 
4 
4 
2 
1 
0 

20 


0 

0 

0.05 

0.05 

0.15 

0.20 

0.20 

0.20 

0.10 

0.05 

0 

1.0 


The  data  of  Table  2  are  illustrated  graphically  in  Figure  2,  which  shows  a  histogram  of 
the  distribution  of  20  detection  ranges. 
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Distribution  of  20  detection  ranges 


50  100  150  200  250  300  350  400  450  500  550  600 

Detection  Range  (m) 


Figure  2.  Distribution  of  detection  ranges  for  the  DRT's  data  in  Table  1. 

In  order  to  draw  conclusions  about  the  detection  range  for  the  trial  several  questions 
need  to  be  addressed.  First,  what  kind  of  distribution  do  the  detection  range  data 
follow,  and  second,  how  are  we  to  draw  conclusions  about  the  trials  results?  Should  a 
cumulative  distribution  function  be  generated  directly  from  the  raw  histogram  or 
should  it  be  obtained  after  first  smoothing? 


The  detection  range  can  be  thought  of  as  a  continuous  variable  that  can  assume  any 
value  over  the  range  interval  from  zero  to  the  maximum  of  the  sonar  range  scale.  If  we 
conducted  a  large  number  of  detection  runs,  the  results  should  converge  to  a 
distribution  representative  of  the  population.  In  as  much  as  the  detection  range  is 
determined  by  many  factors,  none  of  which  is  dominant,  a  normal  distribution  is 
assumed  with  its  mean  >  0.  (This  assumption  is  supported  by  evidence  from  many 
previous  trials  that  demonstrate  that  the  detection  ranges  approximate  a  normal 
distribution.)  Hence,  if  the  number  of  runs  for  the  trial  is  statistically  valid,  using  a 
normal  curve  to  approximate  the  histogram  seems  a  reasonable  approach  to  estimating 
the  sonar  performance. 

A  fitted  normal  curve  is  shown  in  Figure  2.  It  is  a  probability  density  function  (pdf)  of 
detection  range  (the  value  in  x-axis  was  normalised  in  deriving  the  standard 
deviation).  Finally,  the  associated  cumulative  distribution  can  be  calculated  from  the 
mean  and  standard  deviation  which,  in  this  example,  are  339  m  and  130  m  respectively. 

Since  in  this  method  only  data  corresponding  to  runs  achieving  hits  are  considered  in 
the  analysis,  the  cumulative  distribution  function  obtained  by  integrating  the  fitted 
normal  curve  over  the  range  interval  always  scales  to  one.  The  curve  must  be  scaled  to 
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reflect  the  true  detection  probability  achieved  during  the  trial  by  multiplying  by  the  Pd 
obtained  by  dividing  the  number  of  hits  by  the  number  of  runs.  For  the  data  in  table  1 
no  misses  were  recorded  so  P,  is  1.  In  addition,  the  range  is  accumulated  in  an  inverse 
order  to  give  the  traditional  detection  probability  curve  for  minehunting  sea  trials 
(Figure  3). 

Each  measured  detection  range  is  regarded  as  a  sample  of  a  normal  population.  When 
the  standard  deviation  of  the  population  is  assumed  to  be  the  sample  standard 
deviation,  for  large  samples,  the  symmetric  100(l-a)%  confidence  interval  for  the 

expectation  of  a  normal  distributed  range  is  x±zal2^=,  where  a  is  the  standard 

deviation  of  the  distribution  and  N  is  the  number  of  samples.  (For  a  95%  confidence 
interval,  z0025  =  1.96  if  N  is  large).  For  small  N,  as  is  the  case  in  this  example, 
confidence  Nervals  should  be  calculated  from  Student's  t-distribution  instead  of  the 
normal  distribution  (for  details  see  Appendix  1).  For  the  data  in  Table  1,  *  =  337  and  <x 
=  122.  Based  on  these  results,  we  have  the  confidence  limits  shown  in  Figure  3. 


Cumulative  Distribution  Function  of  Range 


Figure  3.  The  Y-axis  is  the  'probability  of  detection  Pd  and  the  X-axis  is  the  range  of  detection. 

The  curve  is  a  cumulative  distribution  function  of  the  detection  ranges  scaled  by 
Pd .  The  dashed  lines  are  the  upper  and  lower  90%  confidence  limits  for  the 
example  data  in  Table  1. 
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3.2  Detection  Probability  With  Respect  to  Closest  Point  of  Approach, 
DWT 

The  aim  in  detection  width  trials  is  to  determine  the  distribution  of  detection  ranges 
with  respect  to  the  lateral  displacement  of  a  target  from  the  ship's  track.  This  kind  of 
trial  requires  a  large  number  of  encounters  to  ensure  sufficient  samples  at  all  cross 
track  range  intervals  and  reliability  of  statistical  results. 

The  following  example  describes  a  procedure  for  processing  data  from  detection  width 
trials.  The  CPA  for  each  planned  encounter  is  calculated  based  on  known  ship  and 
target  positions.  The  CPAs  of  all  the  planned  encounters  and  of  those  achieving  hits 
are  grouped  according  to  athwartship  range  interval  as  in  Table  3.  The  probability  of 
detection  is  then  calculated  at  each  range  interval. 


Table  3.  Distribution  of  planned  encounters  and  data  measured  from  a  DWT. 


Athwartship 
distance  (m) 

Number  of 
encounters 

Number  of  Hits 

Detection  Probability 

-300 

4 

0 

0 

-250 

4 

0 

0 

-200 

10 

0 

0 

-150 

9 

3 

0.33 

-100 

11 

2 

0.18 

-50 

27 

7 

0.26 

0 

25 

24 

0.96 

50 

18 

15 

0.83 

100 

19 

17 

0.89 

150 

24 

4 

0.17 

200 

11 

2 

0.18 

250 

9 

0 

0 

300 

5 

0 

0 

350 

8 

0 

0 

Total 

184 

74 

- 

The  data  in  Table  3  are  shown  as  a  histogram  in  figure  4(a).  The  dark  bars  illustrate  the 
distribution  of  the  planned  encounters  with  respect  to  the  CPA.  The  light  bars  show  the 
distribution  of  detections  achieved  with  respect  to  CPA.  For  the  above  example,  results 
for  Pd  will  be  inaccurate  at  large  CPA  because  the  sample  population  is  too  small,  and 

hence  the  reliability  of  the  estimate  is  poor.  This  illustrates  that  in  planning  detection 
width  trials  it  is  important  to  consider  the  distribution  of  encounters  to  ensure 
sufficient  encounters  in  all  range  intervals.  In  order  to  obtain  an  equal  reliability  across 
lateral  range  we  need  to  plan  for  a  uniform  distribution  for  encounters. 

The  histogram  of  Pd  is  shown  in  Figure  4  (b),  and  a  normal  curve  is  fitted.  As 
indicated,  the  distribution  of  measured  Pd  along  the  athwartships  distance  is  not 
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always  normal,  although  it  is  true  in  general  and  particularly  for  large  numbers  of 
samples.  As  for  the  detection  range  trials,  this  Pd  curve  is  not  a  probability  density 
function  of  Pd  but  a  conditional  probability  function  that  gives  the  probability  of 
detection  at  a  certain  CPA.  The  normal  function  is  assumed  in  the  form: 

Pd(x)  =  a-e~(x-M)2/a>2 

which,  in  this  example  gives  a  =  0.98,  ju  =  0  and  co  =  102.37. 


(a) 


-500  -400  -300  -200  -100  0  100  200  300  400  500 


(b) 


Figure  4.  The  probability  of  detection  with  respect  to  athwartships  distance,  (a)  Shows  the 
distributions  of  encounters  and  hits;  (b)  gives  the  histogram  of  Pdand  data  error 
bars;  and  ( c )  demonstrates  fitted  curve  of  Pd  and  the  specification  of  the  DWT  by 
the  trapezoid  or  rectangle. 
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If  we  put  the  specification,  say,  A  =  200  m  and  B  =  0.9,  in  Figure  4  (c)  then  an  aggregate 
detection  width  (the  area  defined  by  Ax  B  in  Ref  [2])  is  illustrated  by  the  trapezoid  or 
rectangle.  In  general,  the  fitted  curve  will  not  satisfy  the  specification  graphically.  To 
quantify  a  measure  against  the  specification,  the  Equal  Area  Principle  of  Politt  1992  [3] 
can  be  applied.  If  the  area  tinder  the  curve  is  greater  than  or  equal  to  the  area  of  the 
trapezoid  or  rectangle,  the  specification  is  met.  In  practice  there  may  be  a  small  bias  in 
the  trials  results  and  a  direct  fitted  curve  may  be  displaced  left  or  right  of  the  centre 
line.  By  ignoring  this  random  bias,  one  can  fit  data  by  setting  the  mean  to  be  zero.  If 
this  bias  is  significant  it  should  be  investigated  further. 

Now,  we  consider  the  confidence  interval  of  Pd  .  The  Pd  curve  is  assumed  in  general  to 
be  a  normal  distribution,  especially  for  large  samples.  As  a  large  number  of  encounters 
will  be  planned  for  the  detection  width  trials,  the  mean  of  the  curve  will  be  close  to 
zero  and  variation  of  the  standard  deviation  will  not  be  significant  as  the  number  of 
samples  varies.  We  are  not  interested  in  the  confidence  intervals  for  the  normal  feature 
itself.  We  do  however  care  about  the  accuracy  of  the  value  of  Pd  in  each  bin  where  the 
binomial  distribution  is  applied.  For  the  bins  at  large  athwartships  distances,  large 
errors  of  Pd  are  expected  due  to  the  small  number  of  samples  for  each  bin.  As  a  result, 
a  large  confidence  interval  will  be  generated  at  the  edges  of  the  histogram  where  there 
are  limited  samples.  This  is  illustrated  by  the  error  bars  in  Figure  4(b).  (Procedures  for 
calculating  the  confidence  interval  are  given  in  Appendix  2.) 

In  this  example,  the  area  required  by  the  specification  is  Ax  B  =  180  m2,  but  the  total 
area  under  the  curve  of  Pd  is  177.6  m2.  The  results  of  this  trial  therefore  indicate  a 

failure. 

Another  consideration  is,  should  we  form  an  upper  confident  limit  as  we  did  for  the 
detection  range  trials?  It  seems  inappropriate  to  assess  results  using  the  area  under  the 
upper  limit,  because  that  area  will  be  large  if  the  error  in  the  estimate  of  Pd  is  large, 
and  the  assessment  yardstick  therefore  inaccurate. 


4.  Results  Comparisons  for  Detection  Range  Trials 

In  this  section,  four  examples  are  demonstrated  to  compare  results  using  the  previous 
method  and  the  method  proposed.  The  curve  generated  by  using  procedures  described 
in  this  paper  is  shown  as  the  thick  line  in  each  figure.  The  upper  and  lower  90% 
confidence  limits  are  determined  according  to  required  confidence  level  and  the 
number  of  samples  using  Table  4  of  Appendix  1.  The  thin  lines  with  dots  are  the  curves 
determined  using  the  methodology  in  Ref  [1]. 
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The  results  from  the  two  methods  basically  match  when  all  runs  achieve  hits  and  Pd  is 
one,  however  the  agreement  between  the  procedures  decreases  as  Pd  reduces.  This  is 
because  the  definition  of  Pd  in  Ref  [1]  is  inadequate  if  the  target  is  not  detected  on  a 
number  of  nms.  The  new  result  is  more  robust  and  more  reliable  statistically  (Figure  5 
and  Figure  6),  and  it  is  less  susceptible  to  aberrations  in  the  curve  due  to  outlying  data 
points.  When  the  difference  between  the  methodologies  becomes  significant  the 
reliability  of  the  new  method  is  crucial  in  determining  the  most  appropriate  procedure. 


Cumulative  Distribution  Function  of  Range 


Figure  4.  Example  1:  Minehunting  sonar  performance  for  the  DRT.  The  curve  calculated  by 
current  method  is  compared  with  one  from  the  previous  method.  Data  used  are 

given  in  Table  1.  There  are  20  runs  and  20  hits,  therefore  Pd  =  1.0.  For  N  =20  and 
90%  confidence  interval,  k=0.39  (see  Appendix  1). 
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Cumulative  Distribution  Function  of  Range 


Figure  6.  Example  2:  Minehunting  sonar  performance  for  the  DRT.  Data  used  are  in  Appendix 
3,  Table  5.  Where  N=26  and  m-26,  Pd  =  1.0.  k=0.34for  90%  confidence  interval. 


550  600  650  700  750  800 

Detection  Range  (m) 

Figure  7.  Example  3:  Minehunting  sonar  performance  for  the  DRT.  Data  used  are  in  Appendix 
3,  Table  6.  When  N=26  and  m~23,  Pd  =  0.88.  k=0.34for  90%  confidence  interval. 
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Figure  8.  Example  4:  Minehunting  sonar  performance  for  the  DRT.  Data  used  are  shown  in 
Appendix  3,  Table  7.  N=24  and  m=17,  hence  Pd  =  0.71.  k=0.35  for  90% 
confidence  interval. 

Figures  7  and  8  illustrate  effect  of  scaling  by  the  Pd  achieved  during  the  trial.  For  figure 
7,  which  corresponds  to  Table  6  in  appendix  3,  23  detections  were  made  in  26  runs, 
hence  Pd  =0.88.  For  the  data  in  Table  7,  17  hits  were  achieved  in  24  runs.  In  this  case 
Pd=0.71  and  the  curves  in  figure  8  are  scaled  accordingly. 


5.  Conclusion 


Two  methodologies  have  been  described  for  evaluation  of  minehunting  sonar 
performance.  The  first,  used  for  detection  range  trials,  is  regarded  as  an  improved 
methodology  over  that  described  by  Thompson  and  Bell  [1].  The  conclusions  drawn 
using  the  new  methodology  are  more  reliable,  and  better  reflect  the  true  performance 
of  the  sonar  and  consequently  a  more  reasonable  judgment  of  the  sonar  performance. 

The  second  methodology  is  applicable  to  evaluation  of  detection  width  trials.  The 
equal  area  principle  is  applied  to  the  processed  data  to  reflect  the  system  requirement. 
The  question  should  an  upper  confident  limit  be  used  remains  open. 
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As  a  large  number  of  encounters  is  needed  in  detection  width  trails  to  ensure  statistical 
reliability,  it  is  suggested  that  the  trials  procedures  outlined  in  the  RAN  Minehunter 
Coastal  Contract  [4]  be  reassessed  in  light  of  this  requirement. 
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Appendix  1 


When  the  sample  size  N  is  less  than  30,  problems  of  confidence  interval  are  based  on 
small  sampling  theory.  Most  sea  trials  belong  to  this  category.  For  detection  range  trials 
the  standard  deviation  of  range  is  generally  unknown.  If  the  mean  n  for  range  follows 

a  normal  distribution  N(n,cx),  then  its  pivot  variable  follows  Student's  f 

s 

distribution  t(N-l),  where  5  is  the  sample  standard  deviation.  As  is  done  with  the 
normal  distribution  for  confidence  intervals  x±zal2-~,  for  small  samples  we 

replace  za/2  (obtained  from  the  normal  distribution)  by  ta/2  (obtained  from  the  t 
distribution),  then  a  99%,  98%,  95%  and  90%  confidence  interval  for  m  can  be  calculated 
by 

JU  =  X±k-S 

The  factor  k  (~tal2  /N),  determined  by  using  Student's  f-distribution  with  v  degrees  of 
freedom,  is  taken  from  table  4  below: 


Table  4.  Confidence  interval  for  the  expectation  of  a  normal  distribution  from  a  small  samples 
with  v  degrees  of  freedom. 
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Appendix  2 


In  a  given  bin,  we  assume  that  m  hits  were  observed  in  n  detection  runs.  It  is  a 
Bernoulli  trial  with  a  binomial  probability  distribution.  If  we  use  p  =  m/N  to 
approximate  the  probability  of  detection,  the  approximate  100(l-a)  %  symmetric 
confidence  interval  is 


P±Za 


n 


P0--P) 

N 


Here  z  /2  is  the  100(l-a/2)th  percentile  with  typical  values  of  1.645,  1.96  and  2.575 
corresponding  to  the  90%,  95%  and  99%  confidence  levels  respectively. 
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